Image processing device, image processing method, and surgical navigation system

ABSTRACT

Provided is an image processing device including a matching unit that performs matching processing between a predetermined pattern on a surface of a 3D model of a biological tissue including an operating site generated on the basis of a preoperative diagnosis image and a predetermined pattern on a surface of the biological tissue included in a captured image during surgery, a shift amount estimation unit that estimates an amount of deformation from a preoperative state of the biological tissue on the basis of a result of the matching processing and information regarding a three-dimensional position of a photographing region which is a region photographed during surgery on the surface of the biological tissue, and a 3D model update unit that updates the 3D model generated before surgery on the basis of the estimated amount of deformation of the biological tissue.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is continuation application of U.S. patent applicationSer. No. 16/314,461, filed on Dec. 31, 2018, which is a U.S. NationalPhase of International Patent Application No. PCT/JP2017/016163 filed onApr. 24, 2017, which claims priority benefit of Japanese PatentApplication No. JP 2016-138035 filed in the Japan Patent Office on Jul.12, 2016. Each of the above-referenced applications is herebyincorporated herein by reference in its entirety.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, a program, and a surgical navigation system.

BACKGROUND ART

A surgical navigation system that supports a surgeon by presenting apositional relationship between an operating site of a patient and atreatment tool at the time of surgery to a surgeon has been developed.For example, in the technique described in Patent Literature 1, a 3Dmodel of the biological tissue is generated based on tomography imagesof a biological tissue including the operating site of the patientacquired before surgery. Then, at the time of surgery, three-dimensionalpositions of the biological tissue and the treatment tool are detectedand a navigation image in which a display indicating the position of thetreatment tool is added to the 3D model is caused to be displayed, forexample, on a display device set in a surgery room. According to thetechnology, since the surgeon can perform surgery while ascertaining adistance between the operating site and the treatment tool, a distancebetween a part in which contact with the treatment tool in thebiological tissue needs to be avoided and the treatment tool, and thelike using the navigation image, smoother surgery can be executed and itis possible to improve the convenience of a surgeon.

CITATION LIST Patent Literature

Patent Literature 1: JP 2013-202313A

DISCLOSURE OF INVENTION Technical Problem

However, the actual position and shape of the biological tissue of apatient at the time of surgery may change from those before surgery dueto changes in the position of the patient or influence and the like oftreatment such as a laparotomy or craniotomy according to the surgery.In other words, the position and shape of the biological tissue of thepatient at the time of surgery does not necessarily coincide with a 3Dmodel generated based on information acquired before surgery. In such acase, according to the technology described in Patent Literature 1, theaccurate position and shape of the biological tissue in conformity witha current state are not reflected in the navigation image, and thepositional relationship between the biological tissue and the treatmenttool is not accurately displayed. Therefore, appropriate navigationcannot be performed, and the convenience of a surgeon may be degraded.

In view of this, the present disclosure proposes a novel and improvedimage processing device, image processing method, program and surgicalnavigation system which can further improve the convenience of a user.

Solution to Problem

According to the present disclosure, there is provided an imageprocessing device including: a matching unit that performs matchingprocessing between a predetermined pattern on a surface of a 3D model ofa biological tissue including an operating site generated on the basisof a preoperative diagnosis image and a predetermined pattern on asurface of the biological tissue included in a captured image duringsurgery; a shift amount estimation unit that estimates an amount ofdeformation from a preoperative state of the biological tissue on thebasis of a result of the matching processing and information regarding athree-dimensional position of a photographing region which is a regionphotographed during surgery on the surface of the biological tissue; anda 3D model update unit that updates the 3D model generated beforesurgery on the basis of the estimated amount of deformation of thebiological tissue.

In addition, according to the present disclosure, there is provided animage processing method including: performing, by a processor, matchingprocessing between a predetermined pattern on a surface of a 3D model ofa biological tissue including an operating site generated on the basisof a preoperative diagnosis image and a predetermined pattern on asurface of the biological tissue included in a captured image duringsurgery; estimating an amount of deformation from a preoperative stateof the biological tissue on the basis of a result of the matchingprocessing and information regarding a three-dimensional position of aphotographing region which is a region photographed during surgery onthe surface of the biological tissue; and updating the 3D modelgenerated before surgery on the basis of the estimated amount ofdeformation of the biological tissue.

In addition, according to the present disclosure, there is provided aprogram that causes a computer to execute an image processing methodincluding: performing matching processing between a predeterminedpattern on a surface of a 3D model of a biological tissue including anoperating site generated on the basis of a preoperative diagnosis imageand a predetermined pattern on a surface of the biological tissueincluded in a captured image during surgery; estimating an amount ofdeformation from a preoperative state of the biological tissue on thebasis of a result of the matching processing and information regarding athree-dimensional position of a photographing region which is a regionphotographed during surgery on the surface of the biological tissue; andupdating the 3D model generated before surgery on the basis of theestimated amount of deformation of the biological tissue.

In addition, according to the present disclosure, there is provided asurgical navigation system including: a microscope unit that photographsa biological tissue including an operating site of a patient duringsurgery, and acquires a captured image with depth information; aposition sensor that detects three-dimensional positions of themicroscope unit, the patient, and a treatment tool; a display devicethat displays a navigation image in which a display indicating aposition of the treatment tool is added to a 3D model of the biologicaltissue; and an image processing device that causes the display device todisplay the navigation image. The image processing device includes amatching unit that performs matching processing between a predeterminedpattern on a surface of the 3D model generated on the basis of apreoperative diagnosis image and a predetermined pattern on a surface ofthe biological tissue included in a captured image during surgery, ashift amount estimation unit that estimates an amount of deformationfrom a preoperative state of the biological tissue on the basis of aresult of the matching processing and information regarding athree-dimensional position of a photographing region which is a regionphotographed during surgery by the microscope unit on the surface of thebiological tissue, a 3D model update unit that updates the 3D modelgenerated before surgery on the basis of the estimated amount ofdeformation of the biological tissue, and a display control unit thatcauses the display device to display the navigation image using theupdated 3D model. Information regarding a three-dimensional position ofthe photographing region is acquired on the basis of a result ofdetection by the position sensor and depth information of a capturedimage by the microscope unit. According to the present disclosure, inregard to a 3D model of a biological tissue including an operating sitegenerated on the basis of information acquired before surgery and acaptured image at the time of actual surgery, matching processingbetween an image representing a pattern on a surface of the 3D model andan image representing a pattern on a surface of the biological tissueincluded in the captured image is performed. Then, an amount ofdeformation from a state of the biological tissue before the surgery isestimated on the basis of a result of the matching processing, and the3D model is updated on the basis of a result of the estimation. As aresult, the updated 3D model reflects the actual position and shape ofthe biological tissue at the time of surgery, and thus if a navigationimage is generated using the updated 3D model, a more accuratenavigation image conforming to an actual situation can be obtained.Therefore, it is possible to further improve the convenience of a userwho performs surgery using the navigation system.

Advantageous Effects of Invention

According to the present disclosure, convenience of a user can befurther improved. Note that the effects described above are notnecessarily limitative. With or in the place of the above effects, theremay be achieved any one of the effects described in this specificationor other effects that may be learned from this specification.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram which shows an example of a configuration of asurgical navigation system according to the present embodiment.

FIG. 2 is a diagram for describing a function of a position sensor.

FIG. 3 is a diagram which shows an example of a preoperative bloodvessel pattern image.

FIG. 4 is a diagram which shows an example of an intraoperative bloodvessel pattern image.

FIG. 5 is a diagram which shows an example of a calculation model ofFEM.

FIG. 6 is a diagram for describing swelling of a brain.

FIG. 7 is a diagram for describing subsidence of a brain.

FIG. 8 is a diagram which schematically shows a state of a human skullbase.

FIG. 9 is a diagram for describing, in a case in which subsidence of abrain occurs, a case in which a fixed point is set in consideration of arelationship between the brain and the spinal cord.

FIG. 10 is a diagram which schematically shows an example of a generalexisting navigation image in a brain surgical operation.

FIG. 11 is a diagram which schematically shows an example of anavigation image according to the present embodiment in a brain surgicaloperation.

FIG. 12 is a flowchart which shows an example of a processing procedureof an image processing method according to the present embodiment.

FIG. 13 is a diagram which shows a configuration of the surgicalnavigation system according to the present embodiment, including adetailed configuration of an observation device.

FIG. 14 is a diagram which shows a state of surgery using the surgicalnavigation system shown in FIGS. 1 and 13 .

MODE(S) FOR CARRYING OUT THE INVENTION

Hereinafter, (a) preferred embodiment(s) of the present disclosure willbe described in detail with reference to the appended drawings. In thisspecification and the appended drawings, structural elements that havesubstantially the same function and structure are denoted with the samereference numerals, and repeated explanation of these structuralelements is omitted.

Note that the description will be made in the following order.

1. Background of the present disclosure

2. Configuration of surgical navigation system

3. Image processing method

4. Configuration example of observation device

5. Application example

6. Supplement

Here, as an example, an embodiment in which a technology according tothe present disclosure is applied to a brain surgical operation will bedescribed in the following description. However, the present disclosureis not limited to such an example. The technology according to thepresent disclosure may also be applied to other types of surgery.

In addition, a user who uses a surgery support system to be describedbelow is described as a surgeon for the sake of convenience in thefollowing description. However, this description does not limit the userusing the surgery support system, and a subject using the surgerysupport system may also be another medical staff member such as anassistant or a nurse.

1. Background of the Present Disclosure

Before describing preferred embodiments of the present disclosure, theinventors will describe the background of the present disclosure to makethe present disclosure clearer.

In a brain surgical operation, surgery is performed while observing anoperating site of a patient in an enlarged manner through a microscopeunit. For such a brain surgical operation, a surgical navigation systemas described in Patent Literature 1 described above has been developed.In the surgical navigation system, a diagnostic image of a patient'sbrain is photographed by using computed tomography (CT) or magneticresonance imaging (MRI) before surgery. Then, a 3D model of the brain isgenerated on the basis of the photographed diagnostic image. At the timeof surgery, three-dimensional positions of a microscope unit, a patient,and a treatment tool are detected by a position sensor, and a navigationimage in which a display indicating a region corresponding to a currentfield of view of the microscope unit or a display indicating a positionof the current treatment tool is added to the 3D model is displayed on adisplay device installed in a surgery room on the basis of a result ofthe detection. According to such a surgical navigation system, since apositional relationship between the operating site (for example, alesion to be resected) and the treatment tool can be displayed in realtime in the navigation image, a surgeon can perform surgery whileascertaining the positional relationship.

For example, the operating site may be hardly distinguishable inappearance from other normal tissues in some cases. In such a case, byreflecting the position of the operating site obtained by a priorexamination of the 3D model, the surgeon can ascertain the positionalrelationship between the operating site and the treatment tool, which isdifficult to discern with the naked eye, on the navigation image, andthus it is very useful. In addition, since positions of blood vesselsinside the brain can also be reflected in the 3D model, positionalrelationships between the blood vessels and the treatment tool can bedisplayed on the navigation image. Therefore, for example, when thesurgeon incises the brain and causes the operating site to be exposed, arisk of accidentally damaging the blood vessels can be reduced byreferring to the navigation image.

However, it is known that a phenomenon known as brain shift in which thebrain, the shape of which is normally maintained by the cranial bone orcerebrospinal fluid, loses support in a craniotomy and the shape thereofchanges occurs in a brain surgical operation. When brain shift occurs, a3D model generated on the basis of information obtained before surgerydiffers from the shape of brain at the time of actual surgery, and thusa 3D model that accurately reflects the brain is not displayed in thenavigation image. Therefore, the positional relationship between theoperating site and the treatment tool and the positional relationshipbetween the blood vessels and the treatment tool as described abovecannot be accurately represented in the navigation image, which makes itdifficult to perform appropriate navigation.

Therefore, as one countermeasure against brain shift, CT photography orMRI photography is performed during surgery (so-called intraoperative CTphotography or intraoperative MRI photography), and a technology forupdating a 3D model displayed in a navigation image on the basis of theobtained diagnostic image has been proposed. According to this method,the 3D model reflecting the shape of the brain after brain shift isdisplayed in the navigation image, and it is possible to performnavigation conforming to the actual situation.

However, in order to perform the intraoperative CT photography or theintraoperative MRI photography, it is necessary to install a dedicatedCT device or MRI device in the surgery room or to arrange the surgeryroom and a CT room or an MRI room side by side, and thus cost forintroduction is high. In addition, in order to perform theintraoperative CT photography or the intraoperative MRI photography, itis necessary to suspend surgery and perform the photography processing,and thus there are concerns that work is complicated, time for surgeryincreases, and a burden patient increases.

Therefore, the inventors have conducted intensive studies on atechnology capable of further improving the convenience of a surgeon byobtaining an appropriate navigation image more easily in a surgicaloperation, particularly in a brain surgical operation. The presentdisclosure was conceived as a result of this. Hereinafter, preferredembodiments of the present disclosure that the inventors have conceivedof will be described.

2. Configuration of Surgical Navigation System

With reference to FIG. 1 , a configuration according to a preferredembodiment of the present disclosure will be described. FIG. 1 is ablock diagram which shows an example of a configuration of a surgicalnavigation system according to the present embodiment.

With reference to FIG. 1 , a surgical navigation system 10 according tothe present embodiment includes a microscope unit 110, a display device120, a position sensor 130, and an image processing device 140.

(Microscope Unit)

The microscope unit 110 is a means for observing an operating site in anenlarged manner. The microscope unit 110 is configured with an imagesensor, an optical system for guiding light from an observation target(observation light) to the image sensor, and the like accommodated in ahousing. The image sensor generates a signal corresponding to theobservation light, i.e., an image signal corresponding to an observationimage, by receiving and photoelectrically converting the observationlight. As described, the microscope unit 110 is an electronic imagingmicroscope unit that electronically photographs images. In addition, themicroscope unit 110 is configured to photograph an image including depthinformation. For example, the microscope unit 110 has an auto focus (AF)function, and the microscope unit 110 can acquire depth information of aposition which is in focus within an angle of view on the basis ofinformation regarding a focal distance.

In the brain surgical operation, a part of the cranial bone of a patientis craniotomized, and the surface of the brain exposed from the openhead is photographed by the microscope unit 110. The microscope unit 110transmits an image signal according to an acquired captured image, thatis, information regarding the captured image, to the image processingdevice 140 (specifically, the matching unit 142, the brain shift amountestimation unit 143, and the display control unit 145 of the imageprocessing device 140 to be described below).

Note that the microscope unit 110 obtains depth information regardingthe basis of a focal distance, but the present embodiment is not limitedto such an example. The microscope unit 110 may be capable ofphotographing a captured image to which the depth information is added,and a configuration thereof may be arbitrary. For example, a distancemeasurement sensor may also be provided in the microscope unit 110. Asthe distance measurement sensor, for example, various types of sensorssuch as time-of-flight type sensors or laser scan type sensors may beused. According to the method, it is possible to measure athree-dimensional position of brain surface blood vessels moreaccurately than in the case of using a focal distance. Alternatively,the microscope unit 110 may be configured as a stereo camera, and depthinformation may also be obtained on the basis of disparity information.However, since a configuration of the microscope unit 110 becomescomplicated in a case in which a distance measurement sensor or a stereocamera is used, the method of using a focal distance is preferable tomore easily obtain depth information.

In addition, in the present embodiment, the microscope unit 110 can besupported by an arm unit. In addition, an actuator is provided in eachjoint unit of the arm unit, an attitude of the arm unit is controlled bythe actuator being driven, and the position and attitude of themicroscope unit 110 can be controlled. A specific configuration of asupport arm device (observation device) that supports the microscopeunit 110 will be described in the following description (4.Configuration example of observation device).

(Display Device)

The display device 120 is installed at a position at which it is visibleto a surgeon in a surgery room. A captured image of an operating sitephotographed by the microscope unit 110 is displayed on the displaydevice 120 by control from the image processing device 140. The surgeonperforms various types of treatments on an operating site whileobserving the operating site according to a captured image displayed onthe display device 120.

Moreover, a navigation image in which a display indicating a position ofa current treatment tool is added to a 3D model of the brain isdisplayed in the display device 120 under a control of the imageprocessing device 140. A surgeon performs surgery while ascertaining apositional relationship between the brain and the treatment tool(specifically, a positional relationship between a blood vessel and thetreatment tool, a positional relationship between an operating site andthe treatment tool, and the like) according to the navigation imagedisplayed on the display device 120. Note that the details of anavigation image will be described below with reference to FIGS. 10 and11 .

A specific configuration of the display device 120 is not limited, andas the display device 120, various known display devices, for example, aliquid crystal display device, an electro-luminescence (EL) displaydevice, or the like, may be applied.

Note that, although only one block showing the display device 120 isillustrated in FIG. 1 for the sake of convenience, a plurality ofdisplay devices 120 may also be provided in the surgical navigationsystem 10. In a case in which the plurality of display devices 120 areused, a captured image and a navigation image of an operating site canbe displayed on separate display devices 120. Alternatively, in a casein which only one display device 120 is used, a display screen of thedisplay device 120 is divided into two areas, and an operating siteimage and a navigation image can be displayed separately in these areas.

(Position Sensor)

The position sensor 130 detects three-dimensional positions of apatient's head, the microscope unit 110, and a treatment tool. FIG. 2 isa diagram for describing a function of the position sensor 130. FIG. 2schematically shows a positional relationship among a patient 201, themicroscope unit 110, a treatment tool 205, and the position sensor 130.

As shown in FIG. 2 , markers 203 and 207 are attached to the microscopeunit 110 and the treatment tool 205. In addition, a marker 211 is alsoattached to the head of the patient 201 via a support member 209. Theposition sensor 130 is configured as a stereo camera, and can detectthree-dimensional positions of markers 203, 207 and 211, that is, thethree-dimensional positions of the head of the patient 201, themicroscope unit 110, and a three-dimensional position of the treatmenttool 205, on the basis of a captured image thereof.

The position sensor 130 transmits information regarding the detectedthree-dimensional positions of the head of the patient 201, themicroscope unit 110, and the treatment tool 205 to the image processingdevice 140 (specifically, a matching unit 142, a brain shift amountestimation unit 143, and a display control unit 145 of the imageprocessing device 140 to be described below).

Note that a configuration of the position sensor 130 is not limited tosuch an example, and various types of known sensors capable of detectinga three-dimensional position of an object may be used as the positionsensor 130. For example, the markers 203, 207, and 211 may be magnets,and a magnetic sensor may be used as the position sensor 130.

(Image Processing Device)

The image processing device 140 performs various types of displaycontrol in the surgical navigation system 10. Hereinafter, functionalunits of the image processing device 140 will be described in detail.The image processing device 140 includes a 3D model generation unit 141,a matching unit 142, a brain shift amount estimation unit 143, a 3Dmodel update unit 144, and a display control unit 145 as the functionalunits thereof. Note that the image processing device 140 can be, forexample, a processor such as a central processing unit (CPU), a graphicsprocessing unit (GPU), or the like, a control board on which memoryelements such as processors and memories are mixed, a generalinformation processing device such as a personal computer (PC), or thelike, on which a processor is mounted. When a processor included in theimage processing device 140 executes an arithmetic operation process inaccordance with a predetermined program, each of the above-describedfunctions can be realized.

In the surgical navigation system 10, diagnostic images of a patient'sbrain are photographed before surgery, and information regarding thesediagnostic images is managed to be accessed by the image processingdevice 140. The 3D model generation unit 141 generates a 3D model of thebrain on the basis of the diagnostic images acquired before surgery.

As a diagnostic image for generating a 3D model, any image used at thetime of diagnosis may be generally applied. However, as will bedescribed below, a 3D model of the brain generated by the 3D modelgeneration unit 141 is used for the navigation image. Therefore, it ispreferable that the diagnostic image be an image in which an operatingsite is clearly represented (that is, a disease emphasis image) suchthat a position of the operating site can be indicated on the 3D model.In addition, as will be described below, in the 3D model of the braingenerated by the 3D model generation unit 141, an image representing apattern of blood vessels (a blood vessel pattern image) is extractedfrom the 3D model, and used in matching processing of blood vesselpattern images. Therefore, it is preferable that the diagnostic image bean image in which the shape of the blood vessels is clearly represented(that is, a blood vessel emphasis image). In the present embodiment, oneor a plurality of diagnostic images can be used in the generation of a3D model in consideration of these points. For example, as thediagnostic image, an image photographed by MRI, magnetic resonanceangiography (MRA), 3D-CT angiography (CD-CTA), 3D-digital subtractionangiography (3D-DSA), single photon emission CT (SPECT), and the likecan be used.

The 3D model generation unit 141 generates a 3D model by convertingtomography images for each composition obtained by each method such asMRI into a 3D model using, for example, a technology such as volumerendering (VR) or surface rendering (SR) following a continuity of eachcomposition. The processing of generating a 3D model using the 3D modelgeneration unit 141 may be performed using various types of knownmethods, and thus the detailed description thereof will be omitted.

The 3D model generation unit 141 provides information regarding agenerated 3D model of the brain to the matching unit 142 and the 3Dmodel update unit 144.

The matching unit 142 calculates a three-dimensional position on thesurface of the brain during surgery by matching feature points of animage representing a blood vessel pattern of the brain surface includedin a 3D model of the brain generated by the 3D model generation unit 141(hereinafter referred to as a preoperative blood vessel pattern image),and an image representing a blood vessel pattern of the brain surfacephotographed by the microscope unit 110 during surgery (hereinafterreferred to as an intraoperative blood vessel pattern image).

Specifically, the matching unit 142 first specifies a photographingdirection of the microscope unit 110 and a craniotomy position on thebasis of information regarding the three-dimensional positions of thehead of the patient 201 and the microscope unit 110, which is providedfrom the position sensor 130. Next, the matching unit 142 extracts brainsurface blood vessels from the 3D model of the brain generated by the 3Dmodel generation unit 141, and generates a two-dimensional image inwhich the brain surface blood vessel is projected in the photographingdirection of the microscope unit 110 at the specified craniotomyposition, thereby generating a preoperative blood vessel pattern image.FIG. 3 is a diagram which shows an example of the preoperative bloodvessel pattern image. As described above, this preoperative blood vesselpattern image can be clearly generated by using a blood vessel emphasisimage as a diagnostic image for generating the 3D model of the brain.

Next, the matching unit 142, on the basis of information regarding acaptured image provided from the microscope unit 110, extracts an imagerepresenting brain surface blood vessels in the open head, that is, anintraoperative blood vessel pattern image, from the captured image. FIG.4 is a diagram which shows an example of the intraoperative blood vesselpattern image.

Then, the matching unit 142 performs matching processing between featurepoints of the generated preoperative blood vessel pattern image andintraoperative blood vessel pattern image. In the matching processing offeature points, for example, various types of known methods such asFAST, FASTX, STAR, SIFT, SURF, ORB, BRISK, MSER, GFTT, HARRIS, Dense,Simple Blob, and SmartAR (registered trademark) can be used. In thesemethods, it is possible to automatically obtain feature points (forexample, as shown in FIGS. 3 and 4 , a branch point X of blood vessels213 and 215, a bending point Y of a blood vessel, and the like), and toevaluate a degree of matching between the two according to a degree ofcoincidence. At this time, since it is expected that there is blurringin distance and angle between the preoperative blood vessel patternimage and the intraoperative blood vessel pattern image due to brainshift, an affine transformation is performed on the preoperative bloodvessel pattern image in a range with a minute magnification and a smallangle to obtain a case in which a degree of coincidence is the highestin the changed range. As a result, brain surface blood vessels of aregion (a photographing region) on the surface of brain exposed from theopen head at the time of actual surgery and photographed by themicroscope unit 110 are correlated with parts of the brain surface bloodvessels of the 3D model generated by the 3D model generation unit 141.That is, a region on the surface of brain corresponding to thephotographing region at the time of actual surgery is specified on thesurface of the 3D model of the brain generated by the 3D modelgeneration unit 141.

Here, since a reflection of light has a thin stripe shape in a case inwhich an elongated convex-shaped object is photographed like a bloodvessel, in a blood vessel pattern image photographed by the microscopeunit 110, a center of one blood vessel is not displayed, and the bloodvessel may be recognized in a shape of being broken into two lines. Inthis case, synthesis processing in which these are regarded as one bloodvessel can be generally performed by the image processing in processesof the various types of known matching processing. However, since theblood vessel has a curved complicated shape, and the reflectiondirection of the light is not uniform, the recognized stripe-shapedportion does not necessary take such a form as to border the bloodvessel, and blood vessels which should originally have been connectedmay be recognized to be interrupted due to the synthesis processingdescribed above. Therefore, in this case, a blood vessel pattern imagein which border of the blood vessel is, so-called, in a dotted lineshape can be obtained as the intraoperative blood vessel pattern image.In such a case, the matching unit 142 determines matching between anintraoperative blood vessel pattern image in a dotted line shape and apreoperative blood vessel pattern image in a solid line based on avector direction and a disposition pattern of each image.

The matching unit 142 provides information regarding a matching result(that is, information regarding the specified region corresponding tothe photographing region at the time of actual surgery on the surface ofthe 3D model of brain) to the brain shift amount estimation unit 143.

Note that, in the above description, as the intraoperative blood vesselpattern image, an image extracted from a captured image by themicroscope unit 110, that is, an image with visible light, is used, butthe present embodiment is not limited to such an example. As theintraoperative blood vessel pattern image, for example, an imagephotographed with an emphasis on blood vessels of indocyanine green(ICG) fluorescence blood angiography or the like. In this case, themicroscope unit 110 includes a function of performing fluorescencephotographing and ICG fluorescence images of blood vessels can beacquired by the microscope unit 110. It is possible to make a featurepoint in the intraoperative blood vessel pattern image clearer, and tocause accuracy of the matching processing to be improved by using suchan image photographed with an emphasis on blood vessels.

The brain shift amount estimation unit 143 estimates an amount ofdeformation of brain caused by craniotomy from a preoperative state onthe basis of a result of the matching processing by the matching unit142. In the present embodiment, the brain shift amount estimation unit143 estimates the amount of deformation of brain using a finite elementmethod (FEM). Specifically, the brain shift amount estimation unit 143,first, generates a calculation model of the FEM by defining athree-dimensional mesh at each vertex of voxel of the 3D model of braingenerated by the 3D model generation unit 141. FIG. 5 is a diagram whichshows an example of a calculation model of the FEM. In the example shownin FIG. 5 , a calculation model 217 of the FEM is generated by setting amesh 219 in a regular tetrahedral shape with respect to the 3D model ofbrain.

Next, the brain shift amount estimation unit 143 sets an initialcondition and a boundary condition for the FEM calculation. First,initial condition will be described. Specifically, the brain shiftamount estimation unit 143 sets an amount of displacement of brain inthe open head before surgery on the basis of a result of the matchingprocessing by the matching unit 142 as the initial condition.

In the processing of setting the amount of displacement, the brain shiftamount estimation unit 143 calculates three-dimensional coordinates inaccordance with a current body position of the patient 201 in a regioncorresponding to a photographing region on the surface of the 3D modelof brain in a case in which it is assumed that brain is not deformed arecalculated on the basis of a result of the matching processing andinformation regarding a three-dimensional position of the head of thepatient 201 provided from the position sensor 130. The three-dimensionalcoordinates are, that is, three-dimensional coordinates of each mesh ofthe region corresponding to the photographing region on a surface of thecalculation model 217 in a case in which it is assumed that brain is notdeformed.

On the other hand, the brain shift amount estimation unit 143 calculatesactual three-dimensional coordinates of the surface of the brain in thephotographing region on the basis of information regarding thethree-dimensional positions of the head of the patient 201 and themicroscope unit 110, which is provided from the position sensor 130, anddepth information included in information regarding a captured imageprovided from the microscope unit 110.

A difference of these three-dimensional coordinates is the amount ofdisplacement of brain in the photographing region, that is, the amountof displacement of brain in the open head. Since the brain shift amountestimation unit 143 obtains the amount of displacement of brain in theopen head by calculating the difference, and sets a state in which amesh of a position corresponding to the open head of the calculationmodel 217 is moved by the amount of displacement as the initialcondition of the FEM calculation.

Here, it is generally known that there are two types of brain shift atthe time of craniotomy. They are brain swelling which can occur in anearly stage of craniotomy and brain subsidence which can occur after anelapse of 10 minutes or more since the craniotomy. FIG. 6 is a diagramfor describing brain swelling. FIG. 7 is a diagram for describing brainsubsidence. FIGS. 6 and 7 schematically show a cross-section of thebrain 221 at the middle.

In a case in which swelling occurs as shown in FIG. 6 , the brain shiftamount estimation unit 143 expresses a displacement of a mesh positionedat an open head 223 among meshes positioned on the surface of thecalculation model 217 by extending a distance between respective pointsin these meshes. On the other hand, in a case in which subsidence occursas shown in FIG. 7 , the brain shift amount estimation unit 143expresses a displacement of the mesh positioned at the open head 223among the meshes positioned on the surface of the calculation model 217by reducing a distance between respective points in these meshes.

Next, a boundary condition will be described. The brain shift amountestimation unit 143 sets a mesh which is in contact with cranial bone,dura, or the like and cannot physically move from an anatomical point ofview as a fixed point with respect to the calculation model 217 as aboundary condition. A specific content of the boundary condition differsdepending on an aspect (swelling or subsidence) of brain shift describedabove.

In a case in which swelling occurs as shown in FIG. 6 , a region otherthan the open head 223 in the brain 221 is fixed by the cranial bone andthe dura, and it is considered that a displacement due to the swellingconverges on the open head 223. Therefore, in this case, the brain shiftamount estimation unit 143 sets a mesh positioned in a region other thanthe open head 223 among the meshes positioned on the surface of thecalculation model 217 as a fixed point which is not deformed as theboundary condition. In FIG. 6 , a region corresponding to a mesh treatedas a fixed point on the surface of the brain 221 is indicated by a boldline in a simulative manner.

On the other hand, in a case in which subsidence occurs as shown in FIG.7 , it is considered that a region below a contact point between anouter peripheral surface of the brain 221 and a vertical plane (shown bya dotted line in FIG. 7 in a simulative manner) showing a verticaldirection which is a direction in which gravity acts is fixed by thecranial bone, and a region above it is displaced. Therefore, in thiscase, the brain shift amount estimation unit 143 sets a mesh positionedbelow the contact with the vertical plane among the meshes positioned onthe surface of the calculation model 217 as a fixed point which is notdeformed. In FIG. 7 , in the same manner as in FIG. 6 , a regioncorresponding to a mesh treated as a fixed point on the surface of thebrain 221 is indicated as a bold line in a simulative manner.

Alternatively, in a case in which subsidence occurs, another fixed pointmay also be set in consideration of a relationship between the brain 221and a surrounding biological tissue (for example, spinal cords, bloodvessel, nerves, and the like). Specifically, as shown in FIG. 8 , largeor small holes through which spinal cords, nerves, blood vessels, andthe like pass are present in a human skull base. FIG. 8 is a diagramwhich schematically shows a state of the human skull base. FIG. 8schematically shows a state in which a cranial cavity of human ishorizontally cut and the human skull base is viewed from the above. Atop of the page corresponds to the front.

As shown in FIG. 8 , for example, a large occipital hole 227 throughwhich spinal cords pass, a burst hole 229 through which carotid arteriespass, and the like are present in a skull base 225. Since these spinalcords and carotid arteries are joined to surrounding brain tissues byconnective tissues, a part of the brain 221 which is joined to spinalcords and the like can be regarded as a fixed point which is notdeformed. Accordingly, the brain shift amount estimation unit 143 mayalso set, in addition to the mesh positioned below the contact with thevertical plane, a mesh positioned at a place which is opposed to a holethrough which spinal cords or the like are inserted and is assumed to bejoined to the spinal cords and the like among meshes constituting thecalculation model 217, and a mesh positioned between these meshes asfixed points which are not deformed.

FIG. 9 is a diagram for explaining a case in which a fixed point is seteven in consideration of a relation between the brain 221 and the spinalcords, and the like in the case in which subsidence of the brain 221occurs. FIG. 9 schematically shows a cross section of the brain 221taken along midline similarly to FIGS. 6 and 7 . In addition, in FIG. 9, similarly to FIGS. 6 and 7 , a region corresponding to the meshestreated as fixed points on the surface of the brain 221 are indicated bya bold line in a simulative manner.

In the example shown in FIG. 9 , similarly to the case shown in FIG. 7 ,a mesh positioned below a contact point between the outer peripheralsurface of the brain 221 and a vertical plane (shown by a dotted line inFIG. 9 in a simulative manner) is treated as a fixed point which doesnot deform. In the shown example, furthermore, a fixed point 231 whichis a point fixed by being connected with a spinal cord, and the like onthe surface of the brain 221 is added in consideration of a shape of theskull base 225 as described above. Then, a mesh positioned between thefixed point 231 and the contact point between the outer peripheralsurface of the brain 221 and the vertical plane on the surface of thebrain 221 is treated as a fixed point. In this manner, it is possible toset a boundary condition more suitable for an actual situation bysetting a fixed point from an anatomical point of view in considerationof a shape of the cranial bone.

After setting the initial condition and the boundary condition, thebrain shift amount estimation unit 143 calculates an amount ofdeformation of the entire calculation model 217, that is, an amount ofbrain shift, by an FEM. In such an FEM calculation, for example, brain221 is treated as an elastic body and the amount of deformation of thecalculation model 217 is calculated under the initial condition andboundary condition described above by using a stress-strain relationalexpression for an elastic body. For a ratio of Young's modulus toPoisson, physical property values of the brain 221 obtained from varioustypes of literatures, and the like may be used. Since the FEMcalculation can be performed by various known methods, a description ofspecific processing in the FEM calculation will be omitted.

Here, the brain shift amount estimation unit 143 calculates an amount ofthree-dimensional movement of a brain surface blood vessel in the openhead 223 on the basis of a result of the matching processing by thematching unit 142 and the information regarding three-dimensionalpositions of the head of the patient 201 and the microscope unit 110which are provided from the position sensor 130. Then, a shape of thedeformed calculation model 217 is obtained such that a differencebetween the calculated amount of three-dimensional movement of a brainsurface blood vessel in the calculated open head 223 and an amount ofmovement of a mesh positioned at the open head 223 is a minimum in theFEM calculation. The amount of displacement of each mesh 219 of thecalculation model 217 in this state is the amount of movement of eachvoxel of the 3D model of the brain 221, that is, the amount of brainshift of the brain 221.

The brain shift amount estimation unit 143 provides informationregarding the obtained amount of brain shift, that is, the amount ofmovement of each voxel of the 3D model of the brain 221, to the 3D modelupdate unit 144.

The 3D model update unit 144 updates the 3D model of the brain 221generated by the 3D model generation unit 141 on the basis of theinformation regarding the amount of brain shift obtained by the brainshift amount estimation unit 143. More specifically, the 3D model updateunit 144 causes the 3D model to be deformed by moving each voxel of the3D model of the brain 221 by an amount corresponding to the amount ofbrain shift obtained by the brain shift amount estimation unit 143. As aresult, a 3D model which reflects the brain shift and in which an actualstate of the brain 221 of the current patient 201 is more accuratelyrepresented is generated.

The 3D model update unit 144 provides information regarding the updated3D model of the brain 221 to the display control unit 145.

The display control unit 145 controls driving of the display device 120and causes the display device 120 to display a captured image of anoperating site by the microscope unit 110. In addition, the displaycontrol unit 145 controls the driving of the display device 120, andcauses the display device 120 to display a navigation image in which adisplay indicating a position of the treatment tool 205 is added to the3D model of the brain 221 updated by the 3D model update unit 144. Notethat, as described above, the captured image and the navigation imagemay be displayed on the same display device 120 or may be displayed ondifferent display devices 120, respectively.

Here, “the display indicating the position of the treatment tool 205”may be an icon indicating the treatment tool 205 or a real image of thetreatment tool 205 photographed by the microscope unit 110. Thenavigation image may be obtained by combining an icon indicating thetreatment tool 205 with the 3D model of the brain 221, or superimposinga real image of the treatment tool 205 on the 3D model of the brain 221.In addition, in the navigation image, a lesion which is the operatingsite may be displayed in an emphasized manner, or a blood vessel or thelike which is a dangerous part to avoid a contact with the treatmenttool may be displayed in the emphasized manner. The emphasis display isperformed, for example, by superimposing and displaying annotationinformation (including symbols, words, and/or images) in an operatingsite or a dangerous part. For example, the annotation information isgiven in advance to the 3D model of the brain 221 generated beforesurgery, and in this emphasis display, the annotation informationreflects the amount of brain shift obtained by the brain shift amountestimation unit 143, and is given to a corresponding part of the 3Dmodel of the brain 221 which is updated by the 3D model update unit 144.As a specific display mode of a navigation image, various modes used fora navigation image in a general surgical navigation system can beapplied.

Here, FIG. 10 is a diagram schematically showing an example of a generalexisting navigation image in a brain surgical operation. FIG. 11 is adiagram schematically showing an example of a navigation image accordingto the present embodiment in a brain surgical operation. In FIGS. 10 and11 , a photographic image photographed by the microscope unit is shownby on the left side, and the navigation image is shown on the rightside. As shown in FIG. 10 , in the existing navigation image, since the3D model of the brain 221 generated on the basis of a diagnostic imageacquired before surgery is used as the navigation image as it is,deviation occurs by the amount of brain shift, a positional relationshipbetween the actual treatment tool 205 and the blood vessels of the brain221, and a positional relationship between an icon 233 indicating thetreatment tool 205 in the navigation image and the blood vessels of the3D model 235 of the brain 221.

On the other hand, as shown in FIG. 11 , in a navigation image accordingto the present embodiment, since the 3D model of brain 221 generatedbefore surgery is updated in accordance with the amount of brain shiftand used for the navigation image, the positional relationship betweenthe treatment tool 205 and the blood vessels of the brain 221 coincideswith the positional relationship between the icon 233 indicating thetreatment tool 205 in a navigation image and the blood vessels of the 3Dmodel 235 of the brain 221 well.

In this manner, according to the present embodiment, the amount of brainshift is estimated, and a navigation image is generated using the 3Dmodel of the brain 221 updated in accordance with the estimated amountof brain shift. Since the navigation image reflects a state of theactual brain 221 of the patient 201 at the time of surgery, a positionalrelationship between each tissue of the brain 221 (for example, anoperating site, blood vessel, or the like.) of the brain 221 and thetreatment tool 205 will be displayed more accurately in the navigationimage. Therefore, it is possible to provide more useful information tosurgeon, which is helpful for surgery.

In addition, at this time, estimation of the amount of brain shift isperformed on the basis of depth information included in a captured imageon the surface of the brain 221 acquired by the microscope unit 110, andinformation regarding the three-dimensional positions of the head of thepatient 201 detected by the position sensor 130, the treatment tool 205,and the microscope unit 110. Performance of surgery while the brain 221is photographed by the microscope unit 110 is generally performed in abrain surgical operation, and detection of the three-dimensionalpositions of the head of the patient 201 and the like by the positionsensor 130 is widely used in an existing surgical navigation system.That is, according to the present embodiment, it is possible to estimatethe amount of brain shift using existing facilities without newly addinga device or the like. Therefore, it is possible to obtain a moreappropriate navigation image more easily.

Moreover, the amount of displacement of the three-dimensional positionon the brain surface in the open head 223 is set as an initial conditionat the time of calculating the amount of brain shift by performing thematching processing of blood vessel pattern images in the presentembodiment. Here, for example, the amount of brain shift cannot beobtained with high accuracy only by measuring the amount of displacementof a certain point on the brain surface in the open head 223. In thepresent embodiment, using a result of the matching processing of bloodvessel pattern images, the amount of displacement on the brain surfaceas a “surface” in the open head 223 is obtained, and the amount of brainshift is estimated using the amount of displacement, and thereby it ispossible to obtain the amount of brain shift more accurately. Therefore,a more accurate navigation image which reflects an actual deformation ofthe brain 221 better is obtained.

Note that the amount of brain shift can change over time during surgery.Therefore, in the present embodiment, a series of processing after thematching processing of blood vessel pattern images described above arerepeatedly executed at a predetermined interval (for example, 90minutes, or the like) during surgery. As a result, since a navigationimage conforming to a current situation is always displayed, a moreaccurate navigation image can be obtained.

Here, in a case in which the accuracy in the estimation of the amount ofbrain shift is lowered, a positional relationship between each tissue ofthe brain 221 and the treatment tool 205 is not accurately reflected ina navigation image using the 3D model of the brain 221 which is updatedin accordance with the amount of brain shift. Performing surgery usingsuch an inaccurate navigation image needs to be avoided from a safetypoint of view. Therefore, in the present embodiment, in a case in whichit is concerned that the accuracy in the estimation of the amount ofbrain shift is low and an appropriate navigation image cannot beobtained, a function for issuing a warning is provided in the surgicalnavigation system 10.

More specifically, the image processing device 140 further includes awarning control unit 146 as its function. The warning control unit 146controls driving of the display device 120 and/or other output devices(not shown), and issues a warning indicating that an appropriatenavigation image cannot be obtained. In addition, in a case in which anavigation image is already displayed on the display device 120, thewarning control unit 146 cancels the display of the navigation imagetogether with a warning (that is, causes the navigation function to bestopped). The specific mode of the warning is not limited, and thewarning may be, for example, a visual and/or audible warning. Forexample, the warning control unit 146 performs a warning by causingcharacters or the like to be displayed on the display device 120.Moreover, for example, the output device may be a bell or a speaker, andthe warning control unit 146 performs a warning by voice via the bell orspeaker.

Processing of determining whether or not an appropriate navigation imageis obtained may be performed by matching unit 142. The matching unit 142calculates a matching error when performing the matching processing asdescribed above. Then, by comparing the matching error with apredetermined threshold value, it is determined whether an appropriatenavigation image can be obtained. If the matching error is larger thanthe predetermined threshold value, matching processing is notappropriately performed, and the accuracy in estimation of the amount ofbrain shift performed on the basis of a result of the matchingprocessing is highly likely to decrease. Therefore, in this case, thematching unit 142 determines that an appropriate navigation image cannotbe obtained. On the other hand, if the matching error is equal to orsmaller than the predetermined threshold value, matching processing isappropriately performed, and the accuracy in estimation of the amount ofbrain shift performed on the basis of a result of the matchingprocessing is highly likely to be reliable. Therefore, in this case, thematching unit 142 determines that an appropriate navigation image can beobtained.

Specifically, an amount of three-dimensional movement before and afterthe craniotomy of brain surface blood vessels used for matching (thatis, an amount of movement of the brain surface before and after thecraniotomy in the open head 223) is used as the matching error in thepresent embodiment. That is, the matching unit 142 performs matchingprocessing, and calculates the amount of three-dimensional movement ofthe brain surface blood vessel in the open head 223 on the basis of aresult of the matching processing and the information regarding thethree-dimensional positions of the head of the patient 201 and themicroscope unit 110 provided from the position sensor 130. Here, it isgenerally said that the amount of brain shift is about 2 to 3 cm.Therefore, the matching unit 142 uses this general amount of brain shiftas a threshold value and compares the amount of three-dimensionalmovement of a brain surface blood vessel in the open head 223 with thethreshold value, thereby performing the determination processingdescribed above.

Note that a matching error is not limited to the example, and thematching error may be calculated according to various evaluationfunctions used as an index indicating an error in matching processing ofgeneral images. In addition, a threshold value for determining whetherthe matching processing is appropriately performed may also beappropriately set in accordance with a type of the matching error.

In a case in which it is determined that an appropriate navigation imageis not obtained as a result of the determination processing, thematching unit 142 provides information of the determination to thewarning control unit 146. The warning control unit 146 which hasacquired the information executes processing of causing the warningdescribed above to be output.

Here, two factors are considered as factors for which the matchingprocessing of blood vessel pattern images is not appropriately performedin the following description. The first factor is a shortage of bloodvessel information during surgery. If defect information isinsufficient, it is difficult to extract feature points, and there is aconcern that matching processing between other parts having similarfeature points can be performed. In addition, the second factor isdeficiency of the brain 221 and large deformation associated with theprogress of treatment.

Therefore, in the surgical navigation system 10, in a case in which awarning is issued, a message that blood vessel information needs to benewly acquired may be issued at the same time as the warning. Inresponse to the message, to complement the blood vessel information, adiagnostic image of the patient 201 is newly acquired during surgery.Then, the 3D model generation unit 141 generates the 3D model of thebrain 221 again on the basis of this diagnostic image newly acquired.Using this 3D model of the brain 221 newly generated, the matchingprocessing of blood vessel pattern images by the matching unit 142 isperformed, and the processing of determining whether or not anappropriate navigation image described above can be obtained isperformed again. If a result of the determination is good, processing ofestimating the amount of brain shift by the brain shift amountestimation unit 143 and processing of updating the 3D model by the 3Dmodel update unit 144 are performed, and a navigation image using thisupdated 3D model can be displayed. In this manner, a safer surgery canbe realized by displaying the navigation image only in a case in whichit is determined that the appropriate navigation image can be obtained.

Note that a diagnostic image of patient 201 during surgery can beobtained by intraoperative MRA photography and/or intraoperative ICGfluorescence blood angiography and the like. In particular, in a case inwhich intraoperative ICG fluorescence blood angiography is used,photographing can be performed without causing the patient 201 on apatient bed to move, and thus it is convenient. Alternatively, in a casein which the matching processing of blood vessel pattern images isdifficult, an angiographic X-ray image is imaged, and matchingprocessing between a preoperative blood vessel pattern image and anintraoperative blood vessel pattern image may also be performed via theangiographic X-ray image such that matching processing between theintraoperative blood vessel pattern image and the angiographic X-rayimage and matching processing between the angiographic X-ray imagingimage and the preoperative blood vessel pattern image (for example, animage obtained by MRI, CT, or the like) are performed respectively.Here, angiographic X-ray photography is a photographing method which iswidely used for observation of blood vessels and blood flows. In theangiographic X-ray photography, firstly, a contrast agent isintravenously injected into the patient 201. At this time, if X-rayvideo observation is performed, it is possible to observe a state inwhich light is gradually bright in order of heart, artery, peripheral,and vein. Here, in the angiographic X-ray photography, a target part ofthe patient 201 is photographed by an X-ray CT device. As a result, itis possible to obtain information regarding such as a three-dimensionalposition of a blood vessel, a blood vessel branch level, a blood flowdirection, blood retention, and the like.

The configuration of the surgical navigation system 10 according to thepresent embodiment has been described above. As described above,according to the present embodiment, since a more accurate diseaseposition can be displayed in a navigation image in accordance with thebrain shift, this can help a surgeon to do an exact procedure.Furthermore, surgery mistakes can be prevented and further safety can besecured by evaluating appropriateness of the navigation image, andurging a navigation stop or 3D model update in accordance with a resultof the evaluation.

Note that, a target on which the matching unit 142 performs the matchingprocessing is a two-dimensional image in the above description, but thepresent embodiment is not limited to this example. For example, thematching unit 142 may perform the matching processing between a 3D modelof a brain surface blood vessel extracted from the 3D model of the brain221 and a 3D model of a brain surface blood vessel extracted using thedepth information from a captured image by the microscope unit 110.

Also, in the above description, the display control unit 145 causes theone or more display devices 120 to display a captured image of theoperating site photographed during surgery by the microscope unit 110and a navigation image. At this time, annotation information (includingsymbols, words and/or images) may also be displayed to be superimposedon the captured image. For example, in the 3D model of brain obtainedbefore surgery, annotation information given to a part to be displayedin an emphasized manner such as an operating site and a dangerous partis superimposed to a part corresponding to a captured image photographedduring surgery as an image of a frame surrounding the part or the like.At this time, on the basis of the amount of brain shift obtained by thebrain shift amount estimation unit 143, in a case in which the part tobe displayed in the emphasized manner in the captured image duringsurgery moves, the annotation information set before surgery is notdisplayed as it is, but may be displayed appropriately in accordancewith a display form and state of the corresponding part in the capturedimage during surgery, such as causing the movement to be followed by adisplay of the annotation information. In addition, the annotationinformation may include a voice, and, for example, in a case in whichthe treatment tool 205 approaches a predetermined region (for example, aregion where attention is desired to be called) in the navigation image,voices for calling attention to the effect may also be issued via anoutput device (not shown).

In the above description, in a case in which the matching error in thematching processing by the matching unit 142 is large, a warning for thefact is issued and a message indicating that new blood vesselinformation needs be acquired (that is, a message to the effect that adiagnostic image needs to be acquired again) has been issued, but thisembodiment is not limited to such an example. For example, instead of orin combination with these pieces of processing, a numerical valueserving as an index indicating a degree of the matching error, anumerical value serving as an index indicating a reliability of thematching processing which can be appropriately calculated from thematching error, or the like may be displayed on the display device 120under the control from the warning control unit 146. By performing sucha display, a surgeon can ascertain an accuracy of the matching errorquantitatively, and thus it is possible to appropriately determinewhether a diagnostic image needs to be acquired again, for example, inaccordance with a surgical procedure or the like. Therefore, in a casein which a surgeon determines that the matching error is not that largeand surgery can be performed safely without necessarily acquiring adiagnostic image again, surgery can continue as it is, and thus smoothersurgery can be realized.

Alternatively, instead of or in addition to the processing of issuing awarning to the effect that the matching error is large and theprocessing of issuing a message to the effect that new blood vesselinformation needs to be obtained, other warnings based on the matchingerror may be displayed on the display device 120 under the control fromthe warning control unit 146. For example, in a case in which regionswhere the matching error occurs are concentrated in a specified part inan image, there is a concern that lesions accompanied by blood vesselrupture, risk of ischemia, and the like may occur at the part.Accordingly, in such a case, a frame enclosing the corresponding partmay be displayed, and a word notifying that a risk is likely to increasein the part may be displayed in a navigation image on the display device120 or a captured image photographed by the microscope unit 110.

Note that a specific device configuration of the image processing device140 is not limited. The image processing device 140 may be configured torealize the above-described functions, and the specific deviceconfiguration may be arbitrary. For example, the image processing device140 may be constituted by one device or may be constituted by aplurality of devices. In the case in which the image processing device140 constituted by a plurality of devices, for example, if the functionsschematically shown by the blocks in FIG. 1 are distributed and mountedin the plurality of devices and the plurality of devices are connectedto communicate with each other to operate in cooperation with eachother, functions similar to those of the image processing device 140 canbe realized.

In addition, a computer program for realizing each of the functions ofthe image processing device 140 illustrated in FIG. 1 can be producedcan installed in a processing device such as a PC. In addition, acomputer-readable recording medium storing such a computer program canalso be provided. The recording medium is, for example, a magnetic disk,an optical disc, a magneto-optical disc, a flash memory, or the like. Inaddition, the computer program may be distributed via, for example, anetwork, without using a recording medium.

3. Image Processing Method

A processing procedure of an image processing method according to thepresent embodiment will be described with reference to FIG. 12 . FIG. 12is a flowchart illustrating an example of the processing procedure ofthe image processing method according to the present embodiment. Notethat each of the processes shown in FIG. 12 corresponds to the processesexecuted by the image processing device 140 of the surgical navigationsystem 10 illustrated in FIG. 1 . Since details of the processes havealready been described in the description of the functionalconfiguration of the surgical navigation system 10, detailed descriptionof each process will be omitted in the following description of theprocessing procedure of the image processing method.

Referring to FIG. 12 , a 3D model of brain is first generated on thebasis of a diagnostic image acquired before surgery (step S101) in theimage processing method according to the present embodiment. Theprocessing shown in step S101 corresponds to the processing executed bythe 3D model generation unit 141 of the image processing device 140shown in FIG. 1 .

Note that the processing shown in step S101 described above can beexecuted before surgery is started. After the processing shown in stepS101 is performed, setting of the surgery is performed. The setting ofthe surgery may specifically be carrying into a surgery room of thepatient 201, craniotomy work for causing an operating site to beexposed, disposition of medical staff and surgery equipment, and thelike. Then, processing of step S103 and the subsequent processing areprocessing executed during the surgery.

If the surgery is started, a preoperative blood vessel pattern image ofthe surface of brain included in the 3D model is next matched to anintraoperative blood vessel pattern image of the surface of brainincluded in a captured image photographed by the microscope unit 110 inthe image processing method according to the present embodiment (stepS103).

Next, it is determined whether the matching error is equal to or smallerthan a threshold value (step S105). The processing shown in step S103and step S105 corresponds to processing executed by the matching unit142 of the image processing device 140 shown in FIG. 1 .

In a case in which it is determined that the matching error is largerthan the threshold value in step S105, matching between the preoperativeblood vessel pattern image and the intraoperative blood vessel patternimage is not appropriately performed, and, as a result, estimation ofthe amount of brain shift is not performed with high accuracy, and it isconsidered that an appropriate navigation image cannot be obtained. Inthis case, a diagnostic image is newly acquired. Then, the 3D model ofbrain is generated again on the basis of the diagnostic image newlyacquired (step S107). Then, the matching processing of a blood vesselpattern image is executed again by returning to step S103. Note thatprocessing of displaying a numerical value serving as an indexindicating a degree of the matching error, a numerical value serving asan index indicating a reliability of the matching processing, or thelike may also be performed before the processing shown in step S107. Inthis case, the procedure may proceed to step S109 in a case in which itis determined that a surgeon performs matching processing withsufficient accuracy, the procedure may proceed to step S107 only in acase in which there is an instruction by the surgeon, and the 3D modelof brain may be generated again on the basis of the diagnostic imagenewly generated.

On the other hand, in a case in which it is determined that the matchingerror is equal to or smaller than the threshold value in step S105, theprocedure proceeds to step S109. In step S109, the amount of brain shiftis estimated on the basis of a result of the matching processing of ablood vessel pattern image. The processing shown in step S109corresponds to processing executed by the brain shift amount estimationunit 143 of the image processing device 140 shown in FIG. 1 .

Next, the 3D model of brain is updated on the basis of the estimatedamount of brain shift (step S111). The processing shown in step S111corresponds to processing executed by the 3D model update unit 144 ofthe image processing device 140 shown in FIG. 1 .

Then, a navigation image in which a display indicating a position of thetreatment tool 205 is added to an updated 3D model is displayed on thedisplay device 120 (step S113). The processing shown in step S113corresponds to processing executed by the display control unit 145 ofthe image processing device 140 shown in FIG. 1 . As described above,the processing procedure of the image processing method according to thepresent embodiment has been described.

4. Configuration Example of Observation Device

A specific configuration example of an observation device included inthe microscope unit 110 described above will be described. FIG. 13 is adiagram which shows a configuration of the surgical navigation system 10according to the present embodiment, including a detailed configurationof an observation device.

With reference to FIG. 13 , the surgical navigation system 10 includesan observation device 5301, a control device 5317, and a display device5319. Note that the display device 5319 corresponds to the displaydevice 120 shown in FIG. 1 as described above, and thus descriptionthereof will be omitted.

Referring to FIG. 8 , the observation device 5301 has the microscopeunit 5303 for enlarging and observe an observation target (an operatingsite of the patient 201), the arm unit 5309 supporting the microscopeunit 5303 at its tip, a base unit 5315 supporting the base end of thearm unit 5309, and a control device 5317 that comprehensively controloperations of the observation device 5301. The microscope unit 5303corresponds to the microscope unit 110 shown in FIG. 1 as describedabove.

The microscope unit 5303 is made up of an approximately cylindricalbarrel unit 5305, an imaging unit (not illustrated) provided inside thebarrel unit 5305, and an operating unit 5307 provided in a partialregion on the outer circumference of the barrel unit 5305.

The aperture on the bottom end of the barrel unit 5305 is provided witha cover glass that protects the imaging unit inside. Light from theobservation target (hereinafter, also referred to as observation light)passes through the cover glass and is incident on the imaging unitinside the barrel unit 5305. Note that a light source made up of alight-emitting diode (LED) or the like, for example, may also beprovided inside the barrel unit 5305, and during imaging, light may beradiated from the light source onto the observation target through thecover glass. In a case in which ICG fluorescence blood angiography isperformed, the light source can be configured to emit excitation lightof a predetermined wavelength corresponding to ICG.

The imaging unit is made up of an optical system that condensesobservation light, and an image sensor that senses the observation lightcondensed by the optical system. The optical system is made up of acombination of multiple lenses, including a zoom lens and a focus lens,the optical characteristics of which are adjusted so that an image ofthe observation light is formed on the light-sensitive face of the imagesensor. The image sensor senses and photoelectrically converts theobservation light to thereby generate an image signal corresponding tothe observed image. A sensor capable of color photography including aBayer array, for example, is used as the image sensor. The image sensormay be any of various known types of image sensors, such as acomplementary metal-oxide-semiconductor (CMOS) image sensor or acharge-coupled device (CCD) image sensor. The image signal generated bythe image sensor is transmitted to the control device 5317 as RAW data.At this point, the transmission of the image signal may be conductedfavorably by optical communication. This is because at the surgeryvenue, the surgeon performs surgery while observing the state of theaffected area via the captured image, and thus for safer and morereliable surgery, there is demand for the moving image of the operatingsite to be displayed as close to real-time as possible. Transmitting theimage signal by optical communication makes it possible to display thecaptured image with low latency.

Note that the imaging unit may also include a driving mechanism thatmoves the zoom lens and the focus lens of the optical system along theoptical axis. By suitably moving the zoom lens and the focus lens withthe driving mechanism, the magnification factor of the captured imageand the focal distance during imaging may be adjusted. Also, the imagingunit may be provided with any of various types of functions typicallyprovided in electronic imaging microscope units, such as an autoexposure (AE) function, an AF function or the like.

In addition, the imaging unit may be configured as a so-calledsingle-plate imaging unit having one image sensor, and may also beconfigured as a so-called multi-plate imaging unit having a plurality ofimage sensors. In a case of the multi-plate imaging unit, for example,image signals corresponding to RGB by each image sensor are generated,and a color image may also be obtained by synthesizing these.Alternatively, the imaging unit may also be configured to have a pair ofimage sensors (that is, as a stereo camera) for acquiring image signalsfor a right eye and a left eye corresponding to stereoscopic viewing (a3D display). With a 3D display being performed, a surgeon can moreaccurately ascertain a depth of a biological tissue in an operatingsite. Note that multiple systems of optical systems can be provided tocorrespond to each image sensor in a case of the multi-plate imagingunit.

The operating unit 5307 is constituted by, for example, a 4-directionlever, a switch, or the like, and is an input means that receivesoperation input of the surgeon. For example, the surgeon can input aninstruction to change an enlargement magnification and a focal distanceto an observation target of an observation image via the operating unit5307. When the driving mechanism of the imaging unit appropriately movesthe zoom lens and the focus lens in accordance with the instruction, theenlargement magnification and the focal distance can be adjusted. Inaddition, for example, the surgeon can input an instruction to furtherswitch the operation mode of the arm unit 5309 (an all-free mode and alocked mode, which will be described below) via the operating unit 5307.Note that, in a case in which the surgeon attempts to move themicroscope unit 5303 in the manual mode, an aspect in which the surgeonmoves the microscope unit 5303, holding the barrel unit 5305 is assumed.Thus, it is preferable for the operating unit 5307 to be provided at aposition at which the surgeon can easily operate the operating unit withhis or her finger, holding the barrel unit 5305 so that the surgeon canoperate it while moving the barrel unit 5305.

The arm unit 5309 is configured as a result of multiple links (a firstlink 5313 a to a sixth link 5313 f) being rotatably joined to each otherby multiple joint units (a first joint unit 5311 a to a sixth joint unit5311 f).

The first joint unit 5311 a has an approximately cylindrical shape, andon the leading end (bottom end) thereof supports the top end of thebarrel unit 5305 of the microscope unit 5303, so as to allow rotationabout a rotation axis (first axis O1) parallel to the central axis ofthe barrel unit 5305. Herein, the first joint unit 5311 a may beconfigured so that the first axis O1 is aligned with the optical axis ofthe imaging unit of the microscope unit 5303. Consequently, rotating themicroscope unit 5303 about the first axis O1 makes it possible to changethe field of view as though rotating the captured image.

The first link 5313 a securely supports the first joint unit 5311 a onthe leading end thereof. Specifically, the first link 5313 a is anapproximately L-shaped rod-like member, the leading edge of whichextends in a direction orthogonal to the first axis O1, while also beingconnected to the first joint unit 5311 a so that the end of that edgeabuts the top end on the outer circumference of the first joint unit5311 a. The second joint unit 5311 b is connected to the end of the baseedge of the approximate L-shape of the first link 5313 a.

The second joint unit 5311 b has an approximately cylindrical shape, andon the leading end thereof supports the base end of the first link 5313a, so as to allow rotation about a rotation axis (second axis O2)orthogonal to the first axis O1. The leading end of the second link 5313b is securely connected to the base end of the second joint unit 5311 b.

The second link 5313 b is an approximately L-shaped rod-like member, theleading edge of which extends in a direction orthogonal to the secondaxis O2, while the end of that edge is securely connected to the baseend of the second joint unit 5311 b. The third joint unit 5311 c isconnected to the base edge of the approximate L-shape of the second link5313 b.

The third joint unit 5311 c has an approximately cylindrical shape, andon the leading end thereof supports the base end of the second link 5313b, so as to allow rotation about a rotation axis (third axis O3)orthogonal to both the first axis O1 and the second axis O2. The leadingend of the third link 5313 c is securely connected to the base end ofthe third joint unit 5311 c. By rotating the configuration on theleading-end side, including the microscope unit 5303, about the secondaxis O2 and the third axis O3, the microscope unit 5303 may be moved tochange the position of the microscope unit 5303 on the horizontal plane.In other words, controlling the rotation about the second axis O2 andthe third axis O3 makes it possible to move the field of view of thecaptured image on a flat plane.

The third link 5313 c is configured to have an approximately cylindricalshape on the leading end side, and on the leading end of the cylindricalshape, the base end of the third joint unit 5311 c is securely connectedso that both have approximately the same central axis. The base end sideof the third link 5313 c has a rectangular column shape, and the fourthjoint unit 5311 d is connected to the end thereof.

The fourth joint unit 5311 d has an approximately cylindrical shape, andon the leading end thereof supports the base end of the third link 5313c, so as to allow rotation about a rotation axis (fourth axis O4)orthogonal to the third axis O3. The leading end of the fourth link 5313d is securely connected to the base end of the fourth joint unit 5311 d.

The fourth link 5313 d is a rod-like member that extends approximatelylinearly in a direction orthogonal to the fourth axis O4, while alsobeing securely connected to the fourth joint unit 5311 d so that theleading end abuts the side face of the approximately cylindrical shapeof the fourth joint unit 5311 d. The fifth joint unit 5311 e isconnected to the base end of the fourth link 5313 d.

The fifth joint unit 5311 e has an approximately cylindrical shape, andon the leading end side thereof supports the base end of the fourth link5313 d, so as to allow rotation about a rotation axis (fifth axis O5)parallel to the fourth axis O4. The leading end of the fifth link 5313 eis securely connected to the base end of the fifth joint unit 5311 e.The fourth axis O4 and the fifth axis O5 are rotation axes enabling themicroscope unit 5303 to be moved in the vertical direction. By rotatingthe configuration on the leading-end side, including the microscope unit5303, about the fourth axis O4 and the fifth axis O5, the height of themicroscope unit 5303, or in other words the distance between themicroscope unit 5303 and the observation target, may be adjusted.

The fifth link 5313 e is made up of a combination of a first memberhaving an approximate L-shape with one edge extending in the verticaldirection while the other edge extends in the horizontal direction, anda rod-like second member that extends vertically downward from the unitof the first member that extends in the horizontal direction. The baseend of the fifth joint unit 5311 e is securely connected near the topend of the unit of the first member that extends in the verticaldirection of the fifth link 5313 e. The sixth joint unit 5311 f isconnected to the base end (bottom end) of the second member of the fifthlink 5313 e.

The sixth joint unit 5311 f has an approximately cylindrical shape, andon the leading end side thereof supports the base end of the fifth link5313 e, so as to allow rotation about a rotation axis (sixth axis O6)parallel to the vertical direction. The leading end of the sixth link5313 f is securely connected to the base end of the sixth joint unit5311 f.

The sixth link 5313 f is a rod-like member that extends in the verticaldirection, with the base end securely connected to the top face of thebase unit 5315.

The allowable rotation range of the first joint unit 5311 a to the sixthjoint unit 5311 f is suitably set so that the microscope unit 5303 iscapable of desired motion. Consequently, in the arm unit 5309 having theconfiguration described above, three degrees of translational freedomand three degrees of rotational freedom, for a total of six degrees offreedom, may be realized for the motion of the microscope unit 5303. Inthis way, by configuring the arm unit 5309 so that six degrees offreedom are realized for the motion of the microscope unit 5303, itbecomes possible to freely control the position and the attitude of themicroscope unit 5303 within the movable range of the arm unit 5309.Consequently, it becomes possible to observe an operating site from anyangle, and surgery may be executed more smoothly.

Note that the configuration of the arm unit 5309 illustrated in thediagram is merely one example, and factors such as the number and theshapes (lengths) of the links constituting the arm unit 5309, as well asthe number and arrangement of the joint units and the directions of therotation axes may be designed suitably so that the desired degrees offreedom may be realized. For example, as described above, to move themicroscope unit 5303 freely, the arm unit 5309 preferably is configuredto have six degrees of freedom, but the arm unit 5309 may also beconfigured to have more degrees of freedom (in other words, redundantdegrees of freedom). When redundant degrees of freedom exist, in the armunit 5309, it becomes possible to change the attitude of the arm unit5309 while keeping the position and the attitude of the microscope unit5303 in a locked state. Accordingly, for example, control for higherconvenience such as controlling the attitude of the arm unit 5309 suchthat the arm unit 5309 does not interfere with a view of a surgeonlooking at the display device 5319 can be realized by the surgeon.

Herein, the first joint unit 5311 a to the sixth joint unit 5311 f maybe provided with actuators equipped with a driving mechanism such as amotor, an encoder that detects the rotation angle in each joint unit,and the like. In addition, by having the control device 5317 suitablecontrol the driving of each actuator provided for the first joint unit5311 a to the sixth joint unit 5311 f, the attitude of the arm unit5309, or in other words the position and the attitude of the microscopeunit 5303, may be controlled. Specifically, the control device 5317 mayascertain the current attitude of the arm unit 5309 and the currentposition and attitude of the microscope unit 5303 on the basis ofinformation regarding the rotation angle of each joint unit detected bythe encoder. The control device 5317 calculates a control value (forexample, a rotation angle, generated torque, or the like) for each jointunit to realize the movement of the microscope unit 5303 in accordancewith an operation input from the surgeon using these pieces ofascertained information, and causes a drive mechanism of each joint unitto be driven in accordance with the control value. Note that, at thattime, a control method of the arm unit 5309 by the control device 5317is not limited, and various known control methods such as force controlor position control may be applied.

For example, a surgeon appropriately performs an operation input via aninput device (not shown), and thereby the driving of the arm unit 5309may be appropriately controlled by the control device 5317 in accordancewith the operation input, and the position and the attitude of themicroscope unit 5303 may be controlled. With this control, after themicroscope unit 5303 is caused to move from an arbitrary position to anarbitrary position, it can be fixedly supported at the position afterthe movement. Note that, as the input device, it is preferable to applyan operable device even if the surgeon has a treatment tool in his orher hand, such as a foot switch, considering the convenience of thesurgeon. In addition, the operation input may be performed in anon-contact manner on the basis of gesture detection or sight linedetection using a wearable device or a camera provided in a surgeryroom. As a result, a surgeon belonging to the clean area can alsooperate devices belonging to filthiness at a higher degree of freedom.Alternatively, the arm unit 5309 may be operated in a so-calledmaster-slave manner. In this case, the arm unit 5309 may be remotelycontrolled by the surgeon via an input device installed at a locationremote from a surgery room.

In a case in which the force control is applied, a so-called powerassist control in which the actuators of the first joint unit 5311 a tothe sixth joint unit 5311 f are driven so that the external force fromthe surgeon is received and the arm unit 5309 moves smoothly accordingto the external force may be performed. As a result, when the surgeongrasps the microscope unit 5303 and attempts to move the positiondirectly, it is possible to move the microscope unit 5303 with arelatively light force. Therefore, it is possible to move the microscopeunit 5303 more intuitively using a simpler operation, and to improve theconvenience of the surgeon.

In addition, the driving of the arm unit 5309 may be controlled suchthat a pivoting operation is performed. Here, the pivoting operation isan operation of moving the microscope unit 5303 so that the optical axisof the microscope unit 5303 always faces a predetermined point on thespace (hereinafter, referred to as a pivot point). According to thepivoting operation, it is possible to observe the same observationposition from various directions, and thus more detailed observation ofan affected area is possible. Note that, in a case in which themicroscope unit 5303 is configured such that a focal distance thereofcannot be adjusted, it is preferable that a pivoting operation isperformed in a state in which a distance between the microscope unit5303 and the pivot point is in a locked state. In this case, thedistance between the microscope unit 5303 and the pivot point may beadjusted to a fixed focal distance of the microscope unit 5303. As aresult, the microscope unit 5303 moves on a hemisphere having a radiuscorresponding to the focal distance centered on the pivot point(schematically shown in FIG. 13 ), and even if the observation directionis changed, a clear captured image will be obtained. On the other hand,in a case in which the microscope unit 5303 is configured such that afocal distance thereof can be adjusted, a pivoting operation may beperformed in a state in which the distance between the microscope unit5303 and the pivot point is variable. In this case, for example, thecontrol device 5317 may calculate the distance between the microscopeunit 5303 and the pivot point on the basis of the information regardingthe rotation angle of each joint unit detected by the encoder, and mayadjust automatically the focal distance of the microscope unit 5303 onthe basis of a result of the calculation. Alternatively, according to acase in which an AF function is provided in the microscope unit 5303,the focal distance adjustment may be automatically performed by the AFfunction every time the distance between the microscope unit 5303 andthe pivot point changes due to the pivoting operation.

In addition, the first joint unit 5311 a to the sixth joint unit 5311 fmay also be provided with brakes that restrain rotation. The operationof such brakes may be controlled by the control device 5317. Forexample, when it is desirable to lock the position and the attitude ofthe microscope unit 5303, the control device 5317 applies the brake oneach joint unit. As a result, the attitude of the arm unit 5309, or inother words the position and the attitude of the microscope unit 5303,may be locked without driving the actuators, and power consumption maybe reduced. When it is desirable to move the position and the attitudeof the microscope unit 5303, it is sufficient for the control device5317 to release the brake on each joint unit and drive the actuatorsaccording to a predetermined control scheme.

Such a brake operation input may be performed in response to operationperformed by the surgeon via the operating unit 5307 described above.When the user wants to move the position and the attitude of themicroscope unit 5303, the surgeon operates the operating unit 5307 torelease the brake on each joint unit. As a result, the operation mode ofthe arm unit 5309 switches to a mode allowing each joint unit to berotated freely (all-free mode). Meanwhile, when the surgeon wants tolock the position and the attitude of the microscope unit 5303, the useroperates the operating unit 5307 to apply the brake on each joint unit.As a result, the operation mode of the arm unit 5309 switches to a modein which the rotation of each joint unit is restrained (locked mode).

The control device 5317 generally controls an operation of the surgicalnavigation system 10 by controlling operations of the observation device5301 and the display device 5319. For example, the control device 5317controls driving of the arm unit 5309 by causing actuators of the firstjoint unit 5311 a to the sixth joint unit 5311 f to operate according toa predetermined control method. In addition, for example, the controldevice 5317 changes an operation mode of the arm unit 5309 bycontrolling brake operations of the first joint unit 5311 a to the sixthjoint unit 5311 f.

Here, the control device 5317 may include a function of the imageprocessing device 140 shown in FIG. 1 as described above. That is, thecontrol device 5317 has a combination of a function of the imageprocessing device 140 and each function related to driving of theobservation device 5301 described above. Specifically, for example, theimage processing device 140 corresponds to a so-called camera controlunit (CCU) included in the control device 5317, and performs varioustypes of processing related to a display of an image in the surgicalnavigation system 10.

In the processing related to a display of an image, as described withreference to FIG. 1 , processing of causing a captured image to bedisplayed on the display device 5319, processing of causing a navigationimage to be displayed on the display device 5319, and the like may beperformed. For example, in the processing of causing a captured image tobe displayed on the display device 5319, the CCU performs various typesof signal processing on an image signal acquired by an imaging unit ofthe microscope unit 5303 of the observation device 5301 to generateimage data for a display and to cause the image data to be displayed onthe display device 5319. In the above-described signal processing, forexample, various kinds of known signal processing such as developmentprocessing (demosaicing processing), high image quality processing (bandemphasis processing, super resolution processing, noise reduction (NR)processing, and/or camera shake correction processing) and/orenlargement processing (i.e., electronic zoom processing) may beperformed.

Note that the communication between the control device 5317 and themicroscope unit 5303, as well as the communication between the controldevice 5317 and the first joint unit 5311 a to the sixth joint unit 5311f, may be wired communication or wireless communication. In the case ofwired communication, communication using electrical signals may beconducted, or optical communication may be conducted. In this case, thetransmission cable used for wired communication may be configured as anelectrical signal cable, optical fiber, or a composite cable of the two,in accordance with the communication method. Meanwhile, in the case ofwireless communication, it is no longer necessary to lay down atransmission cable inside the operating room, and thus a situation inwhich the movement of medical staff inside the operating room is impededby such a transmission cable may be resolved.

The control device 5317 may be a processor such as a central processingunit (CPU) or a graphics processing unit (GPU), a control board on whicha processor and a storage element such as a memory are both mounted, orthe like. As a result of the processor of the control device 5317operating in accordance with a certain program, the various functionsdescribed above may be realized. Note that, in the example illustratedin the diagram, the control device 5317 is provided as a separate devicefrom the observation device 5301, but the control device 5317 may alsobe unified with the observation device 5301, such as by being installedinside the base unit 5315 of the observation device 5301, for example.Alternatively, the control device 5317 may be made up of multipledevices. For example, by providing a control board or the like in themicroscope unit 5303 and each of the first joint unit 5311 a to thesixth joint unit 5311 f of the arm unit 5309, and communicablyconnecting these control boards to each other, functions similar to thecontrol device 5317 may be realized.

As described above, a specific configuration example of the observationdevice 5301 including the microscope unit 110 has been described. Notethat, in the illustrated configuration example, the observation device5301 has actuators provided in the joint units 5311 a to 5311 f of thearm unit 5309 thereof, and the arm unit 5309 can be driven, but thepresent embodiment is not limited to the example. For example, themicroscope unit 110 may be supported by a so-called balance arm withouthaving a drive mechanism.

5. Application Example

A state of surgery to which the surgical navigation system 10 shown inFIGS. 1 and 13 is applied will be described. FIG. 14 is a diagram whichshows the state of surgery using the surgical navigation system 10 shownin FIGS. 1 and 13 . In FIG. 14 , a state in which a surgeon 5321performs surgery on a patient 5325 on a patient bed 5323 using thesurgical navigation system 10 is schematically shown. Note thatillustration of the control device 5317 (that is, the image processingdevice 140) among constituents of the surgical navigation system 10 isomitted for the sake of simplicity, and the observation device 5301 issimplified and illustrated in FIG. 14 .

As shown in FIG. 14 , an image of an operating site photographed by theobservation device 5301 is enlarged and displayed on the display device5319 installed on a wall surface of a surgery room at the time ofsurgery. The display device 5319 is installed at a position facing thesurgeon 5321, and the surgeon 5321, while observing the state of anoperating site according to an image displayed on the display device5319, performs various types of processing, for example, cutting off ofan affected area, and the like, on the operating site.

Moreover, a navigation image generated by the image processing device140 is displayed on the display device 5319 or another display device(not shown). The surgeon 5321 performs various types of processing on anoperating site while referring to the navigation image. In thenavigation image, for example, a lesion which is difficult to discernusing a naked eye is displayed in an emphasized manner, and a positionalrelationship between a biological tissue including the lesion and atreatment tool is displayed in real time, and thus it is possible toperform surgery more smoothly by referring to the navigation image.

6. Supplement

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

For example, an object on which matching processing is performed is setas an image representing a pattern of a blood vessel on a surface of abrain in the embodiment described above, but a technology of the presentdisclosure is not limited to the example. An image which is subjected tothe matching processing may be an image showing a pattern on the surfaceof a brain, and may also be another pattern other than a blood vessel.For example, matching processing may be performed on an imagerepresenting a pattern of gyrus of brain or sulcus of brain.

In addition, the technology of the present disclosure has been appliedto a brain surgical operation in the embodiment described above, but thetechnology of the present disclosure is not limited to the example. Thetechnology of the present disclosure can be applied to various types ofsurgical operations in which a navigation system is generally used. Forexample, the technology of the present disclosure can be used in varioustypes of surgical operations performed on other sites such aslaparoscopic surgery or endoscopic surgery. Similarly, even in a case inwhich the technology of the present disclosure is applied to these othersurgical operations, a 3D model including a biological tissue includingan operating site is generated, an amount of deformation of thebiological tissue from before surgery is estimated, and a navigationimage is generated using a 3D model updated on the basis of a result ofthe estimation. Then, at this time, a result of matching a predeterminedpattern on a surface of a biological tissue can be used in theestimation processing of the amount of deformation of the biologicaltissue. The predetermined pattern may be a pattern of blood vesselsexisting on the surface of the biological tissue, and may also be apattern of irregularities on the surface of the biological tissueitself.

In addition, the embodiment of the present disclosure on a medicalpurpose has been described in the above description, but the technologyof the present disclosure is not limited to the example. The technologyof the present disclosure may also be used in other fields on moregeneral purposes. For example, the matching processing used in thepresent disclosure can be applied to matching between an existing mapand an aerial photograph newly photographed. For example, it is possibleto extract a difference from a result of the matching processing, and touse the difference at the time of newly creating a map and the like.Alternatively, for example, the matching processing used in the presentdisclosure can be applied to a monitoring system for fields or ricefields in an agriculture field. For example, an image of a field isperiodically (for example, every day) photographed using a fixed camera,and matching processing is performed between an image in the past (forexample, one day before) and a latest image to extract a difference.Growth condition and the like of crops is analyzed on the basis of thedifference, and thereby it is possible to construct a system whichautomatically notifies an appropriate harvest time.

Further, the effects described in this specification are merelyillustrative or exemplified effects, and are not limitative. That is,with or in the place of the above effects, the technology according tothe present disclosure may achieve other effects that are clear to thoseskilled in the art from the description of this specification.

Additionally, the present technology may also be configured as below.

(1)

An image processing device including:

a matching unit that performs matching processing between apredetermined pattern on a surface of a 3D model of a biological tissueincluding an operating site generated on the basis of a preoperativediagnosis image and a predetermined pattern on a surface of thebiological tissue included in a captured image during surgery;

a shift amount estimation unit that estimates an amount of deformationfrom a preoperative state of the biological tissue on the basis of aresult of the matching processing and information regarding athree-dimensional position of a photographing region which is a regionphotographed during surgery on the surface of the biological tissue; and

a 3D model update unit that updates the 3D model generated beforesurgery on the basis of the estimated amount of deformation of thebiological tissue.

(2)

The image processing device according to (1), further including:

a display control unit that causes a display device to display anavigation image in which a display indicating a position of a treatmenttool is added to the updated 3D model.

(3)

The image processing device according to (1) or (2),

in which the biological tissue is a brain,

the predetermined pattern is a pattern of brain surface blood vessels,and

the shift amount estimation unit estimates an amount of brain shiftwhich is an amount of deformation of brain with a craniotomy.

(4)

The image processing device according to any one of (1) to (3),

in which the matching unit specifies a region corresponding to thephotographing region on the surface of the 3D model by performing thematching processing, and

the shift amount estimation unit estimates the amount of deformation onthe basis of information regarding the specified region corresponding tothe photographing region on the surface of the 3D model and informationregarding a three-dimensional position of the photographing region.

(5)

The image processing device according to (4),

in which the shift amount estimation unit sets a state in which a regioncorresponding to the photographing region on the surface of the 3D modelis displaced in accordance with a three-dimensional position of thephotographing region as an initial condition, and estimates the amountof deformation by calculating an amount of deformation of a calculationmodel corresponding to the 3D model using a finite element method.

(6)

The image processing device according to (5),

in which the shift amount estimation unit sets a boundary condition inthe calculation using the finite element method in accordance with arelationship between the biological tissue and another biological tissuein a periphery of the biological tissue.

(7)

The image processing device according to (6),

in which the biological tissue is a brain,

the other biological tissue is a cranial bone or dura mater, and

a region corresponding to a contact part with the cranial bone or thedura mater in the calculation model is set as a fixed point which is notdeformed in the boundary condition.

(8)

The image processing device according to any one of (1) to (7),

in which information regarding a three-dimensional position of thephotographing region is acquired on the basis of depth information addedto a captured image obtained by photographing the photographing region.

(9)

The image processing device according to (8),

in which the depth information is acquired on the basis of informationregarding a focal distance at a time of photographing the photographingregion.

(10)

The image processing device according to (8),

in which the depth information is acquired on the basis of a detectionvalue of a distance measurement sensor.

(11)

The image processing device according to (8),

in which the depth information is acquired on the basis of disparityinformation obtained by a stereo camera.

(12)

The image processing device according to any one of (1) to (11),

in which the matching unit calculates a matching error which is an indexrepresenting accuracy of matching when the matching processing isperformed, and

the image processing device further includes a warning control unit thatcauses a warning to be output in a case in which the matching error isequal to or greater than a predetermined threshold value.

(13)

The image processing device according to (12),

in which the 3D model generation unit generates a 3D model of thebiological tissue again on the basis of a diagnostic image newlyacquired in a case in which the warning is output.

(14)

An image processing method including:

performing, by a processor, matching processing between a predeterminedpattern on a surface of a 3D model of a biological tissue including anoperating site generated on the basis of a preoperative diagnosis imageand a predetermined pattern on a surface of the biological tissueincluded in a captured image during surgery;

estimating an amount of deformation from a preoperative state of thebiological tissue on the basis of a result of the matching processingand information regarding a three-dimensional position of aphotographing region which is a region photographed during surgery onthe surface of the biological tissue; and

updating the 3D model generated before surgery on the basis of theestimated amount of deformation of the biological tissue.

(15)

A program that causes a computer to execute an image processing methodincluding:

performing matching processing between a predetermined pattern on asurface of a 3D model of a biological tissue including an operating sitegenerated on the basis of a preoperative diagnosis image and apredetermined pattern on a surface of the biological tissue included ina captured image during surgery;

estimating an amount of deformation from a preoperative state of thebiological tissue on the basis of a result of the matching processingand information regarding a three-dimensional position of aphotographing region which is a region photographed during surgery onthe surface of the biological tissue; and

updating the 3D model generated before surgery on the basis of theestimated amount of deformation of the biological tissue.

(16)

A surgical navigation system including:

a microscope unit that photographs a biological tissue including anoperating site of a patient during surgery, and acquires a capturedimage with depth information;

a position sensor that detects three-dimensional positions of themicroscope unit, the patient, and a treatment tool;

a display device that displays a navigation image in which a displayindicating a position of the treatment tool is added to a 3D model ofthe biological tissue; and

an image processing device that causes the display device to display thenavigation image,

in which the image processing device includes

a matching unit that performs matching processing between apredetermined pattern on a surface of the 3D model generated on thebasis of a preoperative diagnosis image and a predetermined pattern on asurface of the biological tissue included in a captured image duringsurgery,

a shift amount estimation unit that estimates an amount of deformationfrom a preoperative state of the biological tissue on the basis of aresult of the matching processing and information regarding athree-dimensional position of a photographing region which is a regionphotographed during surgery by the microscope unit on the surface of thebiological tissue,

a 3D model update unit that updates the 3D model generated beforesurgery on the basis of the estimated amount of deformation of thebiological tissue, and

a display control unit that causes the display device to display thenavigation image using the updated 3D model,

in which information regarding a three-dimensional position of thephotographing region is acquired on the basis of a result of detectionby the position sensor and depth information of a captured image by themicroscope unit.

REFERENCE SIGNS LIST

-   10 surgical navigation system-   110 microscope unit-   120 display device-   130 position sensor-   140 image processing device-   141 3D model generation unit-   142 matching unit-   143 brain shift amount estimation unit-   144 3D model update unit-   145 display control unit-   146 warning control unit

The invention claimed is:
 1. An image processing device, comprising: aprocessor configured to: generate a first three-dimensional (3D) modelof a first biological structure based on a first preoperative image;perform a matching process between a first feature point on a surface ofthe first 3D model of the first biological structure and a secondfeature point on a surface of the first biological structure included inan intraoperative image captured by a medical imaging device at a timeof surgery; specify a region of the first 3D model that corresponds to aphotographing region photographed on the surface of the first 3D modelat the time of the surgery, wherein the region is specified based on thematching process; acquire information regarding a three-dimensionalposition of the photographing region based on depth information includedin the intraoperative image; set a state in which the specified regioncorresponding to the photographing region is displaced in accordancewith the three-dimensional position of the photographing region as aninitial condition; estimate an amount of deformation from a preoperativestate of the first biological structure by calculation of an amount ofdeformation of a calculation model corresponding to the first 3D modelby a determined numerical method, wherein the estimation of the amountof deformation of the first biological structure is based on the initialcondition, information regarding the specified region, and theinformation regarding the three-dimensional position of thephotographing region; and update the first 3D model based on theestimated amount of deformation of the first biological structure. 2.The image processing device according to claim 1, wherein the processoris further configured to control a display device to display anavigation image including a treatment tool at a determined position ofthe updated first 3D model.
 3. The image processing device according toclaim 1, wherein the first biological structure is a brain, the firstfeature point indicates a pattern of brain surface blood vessels, andthe processor is further configured to estimate an amount of brain shiftwhich is the amount of deformation of the brain with a craniotomy. 4.The image processing device according to claim 1, wherein the processoris further configured to set a boundary condition in the calculation ofthe amount of deformation by the determined numerical method inaccordance with a relationship between the first biological structureand a second biological structure in a periphery of the first biologicalstructure.
 5. The image processing device according to claim 4, whereinthe first biological structure is a brain, the second biologicalstructure is one of a cranial bone or a dura mater, and a regioncorresponding to a contact part with the one of the cranial bone or thedura mater in the calculation model is set as a fixed point which is notdeformed in the boundary condition.
 6. The image processing deviceaccording to claim 1, wherein the processor is further configured toacquire the depth information based on information regarding a focaldistance at a time of capture of the photographing region.
 7. The imageprocessing device according to claim 1, wherein the processor is furtherconfigured to acquire the depth information based on a detection valueof a distance measurement sensor.
 8. The image processing deviceaccording to claim 1, wherein the processor is further configured toacquire the depth information based on disparity information obtained bya stereo camera.
 9. The image processing device according to claim 1,wherein the processor is further configured to: calculate a matchingerror which is an index that represents accuracy of a match between thefirst feature point and the second feature point, wherein thecalculation of the matching error is based on the matching process; andoutput a warning in a case in which the matching error is one of equalto or greater than a determined threshold value.
 10. The imageprocessing device according to claim 9, wherein the processor is furtherconfigured to: acquire a second preoperative image in a case where thewarning is output; and generate a second 3D model of the firstbiological structure based on the second preoperative image.
 11. Animage processing method, comprising: generating, by a processor, athree-dimensional (3D) model of a biological structure based on apreoperative image; performing, by the processor, a matching processbetween a first feature point on a surface of the 3D model of thebiological structure and a second feature point on a surface of thebiological structure included in an intraoperative image captured by amedical imaging device at a time of surgery; specifying, by theprocessor, a region of the 3D model that corresponds to a photographingregion photographed on the surface of the 3D model at the time of thesurgery, wherein the region is specified based on the matching process;acquiring, by the processor, information regarding a three-dimensionalposition of the photographing region based on depth information includedin the intraoperative image; setting, by the processor, a state in whichthe specified region corresponding to the photographing region isdisplaced in accordance with the three-dimensional position of thephotographing region as an initial condition; estimating, by theprocessor, an amount of deformation from a preoperative state of thebiological structure by calculation of an amount of deformation of acalculation model corresponding to the 3D model by a determinednumerical method, wherein the estimation of the amount of deformation ofthe biological structure is based on the initial condition, informationregarding the specified region, and the information regarding thethree-dimensional position of the photographing region; and updating, bythe processor, the 3D model based on the estimated amount of deformationof the biological structure.
 12. A non-transitory computer-readablemedium having stored thereon, computer-executable instructions which,when executed by a processor, cause the processor to execute operations,the operations comprising: generating a three-dimensional (3D) model ofa biological structure based on a preoperative image; performing amatching process between a first feature point on a surface of the 3Dmodel of the biological structure and a second feature point on asurface of the biological structure included in an intraoperative imagecaptured by a medical imaging device at a time of surgery; specifying aregion of the 3D model that corresponds to a photographing regionphotographed on the surface of the 3D model at the time of the surgery,wherein the region is specified based on the matching process; acquiringinformation regarding a three-dimensional position of the photographingregion based on depth information included in the intraoperative image;setting a state in which the specified region corresponding to thephotographing region is displaced in accordance with thethree-dimensional position of the photographing region as an initialcondition; estimating an amount of deformation from a preoperative stateof the biological structure by calculation of an amount of deformationof a calculation model corresponding to the 3D model by a determinednumerical method, wherein the estimation of the amount of deformation ofthe biological structure is based on the initial condition, informationregarding the specified region, and the information regarding thethree-dimensional position of the photographing region; and updating the3D model based on the estimated amount of deformation of the biologicalstructure.
 13. A surgical navigation system, comprising: a microscopeunit configured to: photograph a biological structure of a patient at atime of surgery; and acquire an intraoperative image with depthinformation at the time of the surgery; a position sensor configured todetect a three-dimensional position of the microscope unit, athree-dimensional position of the patient, and a three-dimensionalposition of a treatment tool; a display device configured to display anavigation image; and an image processing device, wherein the imageprocessing device includes a processor configured to: generate athree-dimensional (3D) model of the biological structure based on apreoperative image; perform a matching process between a first featurepoint on a surface of the 3D model and a second feature point on asurface of the biological structure included in the intraoperative imagecaptured by a medical imaging device at the time of the surgery; specifya region of the 3D model that corresponds to a photographing regionphotographed on the surface of the 3D model at the time of the surgery,wherein the region is specified based on the matching process; set astate in which the specified region corresponding to the photographingregion is displaced in accordance with a three-dimensional position ofthe photographing region as an initial condition; estimate an amount ofdeformation from a preoperative state of the biological structure bycalculation of an amount of deformation of a calculation modelcorresponding to the 3D model by a determined numerical method, whereinthe estimation of the amount of deformation of the biological structureis based on the initial condition, information regarding the specifiedregion, and information regarding the three-dimensional position of thephotographing region; update the 3D model, generated before thepreoperative image was captured, based on the estimated amount ofdeformation of the biological structure; and control the display deviceto display the navigation image including the treatment tool at adetermined position in the updated 3D model, wherein the informationregarding the three-dimensional position of the photographing region isbased on a result of detection by the position sensor and the depthinformation of the intraoperative image.